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they attempted to put policy into practice without compromising their 
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with these tensions to create a successful reform-based mathematics program 
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TENSIONS BETWEEN COMPETING PEDAGOGICAL AND 
ACCOUNTABILITY COMMITMENTS FOR EXEMPLARY TEACHERS OF 

MATHEMATICS IN KENTUCKY' 

Hilda Borko and Rebekah Elliott 
University of Colorado, Boulder 

Abstract 

This paper presents a focused case study of Ann and Kay, a team of exemplary 
elementary teachers, as they worked to modify their mathematics instruction to be 
consistent with the goals of the Kentucky Education Reform Act (KERA) and Kentucky 
Instructional Results Information System (KIRIS, its innovative high-stakes assessment 
system). At the time of our work with Ann and Kay, the mathematics component of 
KIRIS included three types of measures: open response items, multiple choice items, and 
mathematics portfolios (in a research and development phase), which together assessed 
students' xmderstanding of concepts and procedures, as well as their ability to use this 
understanding to solve problems in other disciplines and real life. Ann and Kay's efforts 
to guide students' creation of mathematics portfolios and prepare them for the open 
response item format focused on increased attention to problem solving, mathematical 
communication, and cormections to real world situations. They often foxmd themselves 
faced with tensions and struggles as they attempted to put policy into practice without 
compromising their pedagogical goals and beliefs. In this case study, we discuss how 
they worked with these tensions to create a successful reform-based mathematics 
program in their 4-5 classroom. 



^ We would like to thank other members of the research team: Shelby Wolf, Monette Mclver, Brian 
Stecher, and Sheila Barron, for their comments on the paper. We extend special thanks to the two 
teachers whose story is presented in the paper. These teachers opened their classroom to us and spent 
many hours sharing their ideas about teaching and learning with us. Without their support, this 
project would not have been possible. An earlier version of this paper was presented at die CCSSO 
National Conference on Large Scale Assessment, Colorado Springs, Jxme 1998. 
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Introduction 



For the past five years, Ann and Kay,^ a team of exemplary elementary teachers 
in an urban Kentucky school district, have worked hard to modify their mathematics 
instruction to be consistent with the goals of the Kentucky Education Reform Act 
(KERA). Like many other teachers in the state, their efforts focused on increased 
attention to problem solving, mathematical communication, and connections to real 
world situations. The change process has not been an easy one for these two 
teachers; they often found themselves faced with tensions and struggles as they 
attempted to put policy into practice without compromising their pedagogical goals 
and beliefs. This paper is, in part, the story of how Ann and Kay worked with these 
tensions to create a successful reform-based mathematics program in their 4-5 
classroom. 

Ann, Kay, and other dedicated teachers were not the only ones who struggled 
to make KERA work. The Kentucky Department of Education (KDE) was 
responsible for operationalizing KERA and determining schools' success in attaming 
its goals. Like Ann and Kay, KDE personnel committed to the reform vision 
embodied in KERA found themselves faced with tensions and struggles as they 
worked to create a measurement and accoimtability system that would assess school 
performance against Kentucky's six broad learner goals, while meeting the 
psychometric requirements for a high-stakes assessment system. This paper is also, 
in part, the story of their efforts. 

Finally, this paper is the story of the Kentucky political system that ultimately 
took these struggles out of the hands of educators by mandating a new 
Commonwealth Accountability and Testing System (CATS) to replace KIRIS 
beginning in the 1998-99 school year. This paper focuses primarily on mathematics 
portfolios, as many of the conflicts of the educational reform were fought on the 
battleground of mathematics portfolio assessment. 

KERA, KIRIS, and Mathematics Portfolios 

KIRIS is Kentucky's innovative, high-stakes assessment system. The 
mathematics component of KIRIS was designed to assess students' mathematical 



^ Ann and Kay are pseudonyms. 
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literacy: their understanding of concepts and procedures, as well as their ability to 
use this understanding to solve problems in other disciplines and in real life (KDE, 
1995a). It consisted of three types of assessments, whose status in the Accoimtability 
Index fluctuated over the years. 

• Open response items, consistently part of the Accountability Index, 
provided the opporhmity for students to demonstrate their ability to apply 
skills and show understanding of concepts. 

• Multiple choice items tested both computational and problem-solving skills. 
They were part of original KIRIS testing, dropped, and then reintroduced in 
199^1997; to be included in the Cycle 4 Accoimtability Index. 

• Mathematics portfolios were designed to assess students' comprehensive 
understanding of mathematics and mathematical processes. They were 
included in the Accoimtability Index through the 1995-1996 testing cycle. 

KDE had high expectations for the role of portfolios in KERA. From the 
begirming, they were to serve dual purposes: to provide measures of student 
achievement for the accountability system and to encourage changes in curriculum 
and instruction. Thus, "The mathematics portfolio must become the foundation 
upon which the instructional program is built. It is not and must not be an add-on to 
the instructional program" (KDE, 1995b, p. ii). When used as envisioned. 

Quality portfolio assessment makes it possible for teachers to identify accurately the 
learning needs of individual students and student groups, to review the appropriateness 
of curriculum goals and content, and to evaluate the quaUty of their own teaching. In 
effective learning environments, assessment and instruction are inexorably linked. (KDE, 

1997a, p. 22) 

Despite this central role, mathematics portfolios were removed from the 
Accountability Index in Fall 1996 because of concerns such as reliability of scoring 
and lack of standardization in the way portfolio entries were produced. KDE began 
a three year research and development (R&D) effort to address these concerns. 
Selected teachers worked with KDE, with the goal of reintroducing portfolios into 
the Accountability Index in 2000-2001. Key changes considered during the R&D 
phases included: 

• promoting more concisely written portfolio entries that emphasize 
mathematical representations (e.g., models, charts, graphs, symbols) with 
shorter verbal explanations embedded around the representations; 
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• field testing common sample tasks provided by KDE; and 

• experimenting with various scoring options and associated guidelines. 

To address concerns about lack of standardization, KDE developed a written 
document containing Philosophical Guidelines /Code of Ethics for preparation of 
mathematics portfolios and made these guidelines the focus of the 1998 Spring Math 
Portfolio Training offered to teacher leaders throughout the state. These guidelines 
stressed student ownership of portfolio work: 

Students must have total ownership of their mathematics. Any intervention from 
teachers, peers and/or others should enhance rather than remove or diminish that 
ownership. ... At no time should students' ideas, revisions, or editing be characterized as 
teacher-authored, peer-authored, or parent-authored. (KDE, 1997a, p. 22) 

Further, during the portfolio preparation process. 

Feedback should be given to the students whenever the teacher feels it is appropriate. 
Feedback occurs by conferencing with students individually, group conferences, peer 
reviews, and post-it-notes. It is essential, however, that during these conferences the 
student retain ownership of his/her work. Teachers and/or others will not at any time 
actually solve the problem on portfolio entries, do the mathematics or do the writing. 

The teacher will support the student in self-assessing and making final decisions; 
however, the student will decide what to incorporate and what to reject. (KDE, 1997a, 
p.24) 

Throughout the R&D phase, KDE personnel remained convinced of the value 
of portfolios and committed to having them continue to serve the dual purposes for 
which they were designed. They engaged in tremendous efforts to address the 
concerns expressed by measurement specialists as well as some teachers throughout 
the state. Their efforts were supported by teachers such as Ann and Kay. 

KERA's Impact on Mathematics Instruction 

Arm and Kay are exemplary teachers who have a strong commitment to the 
goals of KERA and KIRIS. And, they are teaching at a school that has embraced 
these goals. Each summer for the past several years, the school's curriculum 
committee has taken on the task of revising its curriculum for one subject area, to 
align it with the appropriate component of Transformations: Kentucky's Curriculum 
Framework. The mathematics curriculum was revised in Summer 1996. As Ann 
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explained, now that the curriculum was realigned "to the state guidelines and to the 
NCTM guidelines, ... we have a much tighter, more comprehensive math program." 

For Ann and Kay, this revamping of the mathematics curriculum was 
accompanied by substantial changes in their mathematics teaching. To teach toward 
KERA goals. 

We had to devote a lot of time to talking about math and talking about solving problems 
and solving problems in different ways. ... Fd say our computation instruction is maybe 
20, 30% of what we do, whereas before, it was expected to be about 80% of what we did. 

So that's flipped completely. 

As a result of these changes in their mathematics program and associated 
changes in their expectations for students, Ann and Kay believe that their students 
have a better imderstanding of mathematics than they did in the past. 

Our kids are better mathematicians because we're listening to what they have to say 
instead of just grading the numbers on a page. I see a lot of kids that have some strengths 
in math that just 20 computation problems would not pick up. 

Their school's 1997 KIRIS scores support this positive assessment. The school 
met its goal for an overall KIRIS score and was classified as "successful" according 
to the state's performance criteria. Further, their mathematics score increased 
approximately 10 points between 1996 and 1997. 

The Dilemma of Hands-On Instruction and Hands-Off Accountability 

The road to success has not been an easy one for Ann and Kay. Although 
strong supporters of KERA and its educational policies and practices, they 
sometimes foimd themselves facing serious challenges as they attempted to revamp 
their mathematics instructional program. Most problematic were the dilemmas they 
confronted when their pedagogical goals and beliefs conflicted with the 
requirements of KIRIS testing. These dilemmas occurred primarily with 
mathematics portfolios — ^the only component of the KIRIS mathematics testing that 
was completed during teachers' ongoing instructional programs. 

Although portfolios were not included in the KIRIS Accoimtability Index for 
the 1996-1997 and 1997-1998 testing cycles, Ann and Kay (along with the other 4-5 
teachers at their school) continued to treat them as an integral part of their 
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mathematics program. And, they continued having their students prepare portfolios 
according to the KIRIS guidelines. 

At the same time, Ann and Kay believed strongly that their students needed 
substantial assistance to learn the mathematics content and processes needed to put 
together high quality portfolio entries. It was in attempting to provide this 
assistance, while still ensuring that students complete a sufficient number of 
independent portfolio entries to meet the accoimtability requirements that their 
pedagogical beliefs came into conflicts with KIRIS. As Ann explained, 

... you really want the kids' independent work [portfolio entries] to be the ones in the 
assessment portfolio. But you've got to train them. You've got to model them. You've got 
to go through the whole thing with them or they'll never be able to do it on their own ... 
not to the state standard, anyway. 

We saw many instances of the outstanding scaffolding that Ann and Kay 
provided to their students during our three visits to their classroom. Some of the 
strongest examples occurred during Daily Oral Mathematics (DOM), when they 
used open response type tasks to help students learn to use appropriate problem 
solving strategies and to write mathematical explanations to accompany their 
problem solutions. 



"Hands-On" Scaffolding of a DOM Activity 

In our Fall 1997 visit, Ann and Kay worked with the fourth-grade students and 
then with the fifth graders on DOM tasks. They separated the groups so that they 
could provide more structure and modeling for the fourth graders. As Ann 
explained, with fourth graders, "we really try to scaffold that experience because ... a 
lot of them have never done any (open-ended problem solving tasks). They don't 
have a clue what strategies are. ... So we spend a lot of the fourth-grade year 
modeling it. With fifth graders, we like to turn them more loose; they need to be 
doing it more independently." 

Fourth graders worked with the following prompt: 

Make a word problem that has 42 as its solution. 

Explain how someone could solve the problem. 
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Ann introduced the problem to the students by paraphrasing it and providing 
fairly specific instructions about what to do. She emphasized: "Is there one right 
answer? No, there are probably at least a million ways to do the problem. Choose 
one way, and write a problem. Then you have to explain how someone would solve 
the problem." Ann then told the students to take out their papers with guidelines for 
Open Response and Portfolio task write-ups. She set the timer for 4 minutes and the 
students began working. As students worked, Ann and Kay walked around the 
room, looking at their papers but not providing assistance. After a couple of minutes 
Kay walked over to the area of the classroom where the fifth-grade students were 
working. 

When the timer went off, Ann asked how many students needed more time. 
Virtually every student put up a hand, and she set the timer for an additional 4 
minutes. When it went off for the second time, Ann and Kay discussed the problem 
with the students. They asked several students to share their answers and focused 
the conversation primarily on the strategies they used, what counts as a good 
strategy, and their use of good mathematical language. 

For example, Ann asked someone to tell what strategy they used. 

Student: Guess-and-check 

Ann: How did you use guess-and-check? 

Student: I guessed first and then checked with addition. 

Kay: What strategy did you use to get the word problem? 

Student did not respond. 

Ann: Read me your word problem. 

Student: 46-4=42. 

Ann: You did a wonderful restatement. I know just what strategy you used. 

Look at our list. What strategy did you use? 

Student: Writing an equation. 

Ann: Writing an equation with what in it? 

Student did not respond. 

Ann: Not addition. 

Student: Subtraction. 
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Arm: That's a good strategy, but it's not guess-and-check. 

After the discussion, Ann put up another overhead with the rubric for scoring 
open response tasks. She asked the students to take out their red pens to self- 
evaluate. They then had several minutes to finish or revise their work based on the 
discussion and to evaluate their completed response using the rubric as a guide. 
They handed in their work so that Ann and Kay could evaluate and score it before 
returning it to them on the next day. The activity took just over 20 minutes. 

This instance of instruction during Daily Oral Mathematics illustrates the 
multi-faceted approach that Arm and Kay used to guide and scaffold students' 
ability to work independently on mathematics problems. It also highlights several 
ways in which these two exemplary teachers attempted to teach toward the goals of 
KERA and KIRIS. Both teachers mentioned repeatedly the importance of helping 
students learn to think, to structure their time, and to take risks in their 
mathematical work. They explained their use of the timer to help students learn to 
structure their time: "Our biggest problem is getting them in a time frame. ... They 
always want extra time. ... especially kids who are afraid to think." So, to make them 
more aware of time "We'll get them started. We'll set it for 4 minutes. ...There's a lot 
of fear so a lot of times they'll sit there for 4 minutes and then we set it for 4 more." 
They also talked about how difficult it was for them to keep quiet and allow 
students to work independently. Their natural tendency was to provide even more 
coaching than they did. Indeed, they joked with each other and with us about 
having to leave the room, put tape on each other's mouths, and take other draconian 
measures to keep quiet as they watched students struggle to complete these tasks at 
the beginning of the year. 

"Hands-Off" Accountability for Portfolio Tasks 

Ann and Kay see portfolios as valuable pedagogical tools, and they have 
incorporated them into their ongoing instructional program. They use at least one 
portfolio activity within each mathematics unit, most often as an assessment at the 
end of the imit. However, they struggle with the question of how to use prompts in 
a way that is pedagogically soimd and will result in completed portfolios that meet 
KIRIS guidelines by the end of the school year. Their struggles center on two 
issues — ^the nature of the prompts and the scaffolding of the task. 
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Arm and Kay agree with the KDE guidelines regarding characteristics of 
appropriate tasks for mathematics portfolio assessment: that they can be approached 
in multiple ways, require more than an algorithmic solution strategy, allow students 
to select and apply a strategy of their choice, require grade-level mathematics, and 
provide a meaningful context to apply skills or concepts (KDE, 1997b). However, in 
their experience, students are not ready to work on tasks with these characteristics at 
the beginning of the school year. Thus, when we visited their classroom in 
September 1997 they had just used a portfolio task at the end of the 
Numeration/Computation unit which was, in their estimation, "very weak," 
focusing more on computation than conceptual understanding or problem solving. 
As Arm explained. 

This is where they are right now and if you want to give them something that's way 
beyond them, number one you're wasting your time and number two you're going to 
lead them to places they're not ready to go. 

Similarly, students are not ready to work independently on portfolio tasks at 
the beginning of the year. They need a lot of "directed, scaffolded training." As part 
of this training Ann and Kay provide students with a template specifying 
components of a written portfolio response and guide students through drafting the 
components, one at a time. As Ann explained. 

When we started them at the beginning of the year, we would write out a template. ... 

We'd give them very clear direction on what a written response would look like. ... 

[Then] we would say, "You do an introduction. I'm going to set the timer for about 5 
minutes. Give me what you think is a good introduction." And then, when the timer 
went off, we'd say, "Would anybody like to read what they've got?" Then the children 
would read some of the things that they [wrote] and we can either say "that's great" or 
"you're on the right track." And the other kids would benefit from that. So we're doing a 
lot of directed instruction. 

Ann and Kay understand well that portfolio tasks scaffolded in this way cannot 
be included in students' assessment portfolios. They "can go in their working 
portfolio. But when it comes down to it, we have to have [students] choose their 
most independent work. Even if they wanted to choose something that we had 
orchestrated to that degree together, I wouldn't feel comfortable about putting it in 
an assessment portfolio." 
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For their class of 1998, Ann estimated that "it's going to take us to January, 
December, before we can get kids to really do the types of things we'd like them to 
be able to do as a fifth grader on a math portfolio." However, given the KDE 
requirement for six entries in a student's mathematics portfolio, Ann and Kay felt 
they did not have until December or January to teach through modeling, scaffolding, 
and providing feedback. Instead, students needed to begin working independently 
on their mathematics portfolios much earlier in the school year. Hence, they faced a 
dilemma as they attempted to use portfolios in a way that was pedagogically sound 
but did not violate guidelines of student ownership. 

With portfolios in a R&D phase, Ann and Kay were able to postpone 
confronting this dilemma head on. The fact that portfolios were not part of the 
Accountability Index "allows you to be a little more flexible and creative. And it's 
allowed us to do more modeling, which the kids really benefited from." As a result, 
Ann reported in Spring 1997, 

I feel really good about the portfolio process that we've had in math throughout the 
school year. I think that having the scaffolded activities and lessons and some of the 
direct feedback we've given them in their writing will help them now that they're in the 
more independent stage. 

Ann and Kay have not resolved this dilemma, and they continue to wonder 
whether portfolios can serve both pedagogical and accountability purposes. Thus, in 
Fall 1998 Ann ruminated, "Is this an instructional tool or is this an assessment tool? I 
think there's still a real big question in my mind as to whether it should be both." 
They believe, first and foremost, that portfolios should be an instructional tool, even 
if that means keeping them out of the Accountability Index. On the other hand, they 
would like to see portfolios be reinstated in the Accountability Index to ensure their 
continued place in teachers' mathematics programs. 

KDE's Continuing Challenges to Address Portfolio Concerns 

Just as Ann and Kay found no easy resolution to their "hands-on, hands-off 
dilemma," KDE personnel did not discover an easy solution to the challenges 
surrounding the mathematics portfolio assessments. They continued to work hard at 
addressing these challenges. Their efforts included the development of materials 
such as booklets of sample portfolio tasks for elementary, middle, and high school 
levels; pamphlets providing guidance in selecting appropriate tasks at each of these 
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levels; and Analytic Piece Scoring Guides for Grades 5, 8, and 12. They conducted a 
series of training sessions focused on characteristics of appropriate portfolio tasks, 
using the new scoring guide and ethical considerations in portfolio scoring, for 
teacher leaders throughout the state in Fall 1997 and Spring 1998. In addition, they 
planned a Summer 1998 working session for the Mathematics Portfolio Advisory 
Group, to score a sample of mathematics portfolios at all levels and to develop 
materials for distribution to teachers across the state. These materials will include 
sample tasks, scoring guides, and benchmark portfolio entries at all levels. 

KDE personnel felt confident that they were making good progress on a 
number of fronts. For example, based on their work with teachers using the Analytic 
Piece Scoring Guide at the Fall 1997 and Spring 1998 Mathematics Portfolio Training 
Sessions, they felt that teachers were becoming more consistent in their scoring of 
portfolios. Also, based on their initial review of the mathematics portfolios 
submitted to be scored at the summer meeting they were very encouraged by the 
quality of the portfolio entries. 

The Kentucky General Assembly Takes Mathematics Portfolios 
Out of Educators' Hands 

Despite the efforts of KDE and many Kentucky teachers, the General Assembly 
passed House Bill 53 at its 1998 Regular Session, effectively taking decisions about 
mathematics portfolios out of educators' hands. House Bill 53 mandates major 
changes in Kentucky's assessment and accountability system. A new 
Commonwealth Accountability and Testing System (CATS) will replace KIRIS 
beginning in the 1998-99 school year. KIRIS was administered as designed in 1997- 
98, and data will be used to compute school scores for the third (and final) 
Accountability Cycle. The new Assessment and Accountability System will be in 
place for the 1998-99 school year, with baseline data collected in 1998-99 and 1999- 
2000, and the first accountability determination to be made after the 2000-2001 and 
2001-2002 school years. 

House Bill 53 mandates that CATS will include: 

• Open response or multiple-choice items, or both, to assess student skills in 
reading, mathematics, science, social studies, the arts, the humanities, and 
practical living and vocational studies; 

• An on demand assessment of writing; 
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• Writing portfolios consisting of samples of student work; and 

• Performance assessment events for schools that have students enrolled in 
competitive performing arts programs. 

There is no mention of mathematics portfolios in the new legislation. The 
superintendent's office has indicated, however, that mathematics portfolios will not 
be part of the new testing system. 



Post-Script 

Ann and Kay are disappointed that mathematics portfolios will not be part of 
the new Commonwealth Accoimtability Testing System. They intend to continue 
including portfolios in their mathematics program. However, they are not optimistic 
about the future of these innovative instructional and assessment tools. As Ann 
indicated in our final interview: 

Now they've come out and said they don't want math as part of the assessment. I'm 
really imcomfortable with that, because ... teachers aren't going to do it [keep portfolios 
in their mathematics programs]. There's no accountability. If there's no accoimtability, 
it's not going to happen. 

We found ourselves watching the demise of KIRIS with great interest. 
Although certainly far from perfect, KIRIS was viewed nationally as perhaps the 
most innovative state assessment system. As noted in Education Week (September 2, 
1997), "KIRIS has been lauded for its use of performance assessments in which 
students must show what they know. It has also been controversial inside and 
outside the state...." Kentucky put enormous resources into developing and 
administering the assessment system, and into professional development 
opportunities for teachers. And KDE involved exemplary teachers like Aim and Kay 
in many aspects of its ongoing work to improve KIRIS and assist teachers in its 
implementation. Yet, these efforts were not enough, and some of the most 
innovative elements of KIRIS will be conspicuously absent from CATS. We are left 
wondering whether Ann is right that changes in classroom practice will disappear. 
We also wonder what it takes to make innovations like KIRIS work. Or is House 
right when he notes that "Teachers know from experience that reformers cannot 
guarantee the promised outcomes of their reforms.... Most reforms are the simple 
ideas of political and educational entrepreneurs. Almost all become fads, only to 
disappear eventually.... Why bother?" (House, 1996, pp. 8-9). 
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